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ABSTRACT 

The National AssesiUent o£ Educational Progress 
(NAEP) approach to developing assessments is described. NAEP will 
siolicit views concerning assessment objectives and use the consensus 
approach. NAEP results can impact educational policy and school and 
classroom practices. Several policies have been adopted to provide 
improved information about specific subject area achievement. Subject 
areas will be selected to address issues of national concern and 
information needs. Each biennial assessment will include four subject 
areas. Subject area assessments will be more closely integrated and 
related and will include reading. Innovative item development will be 
assured through adoption of several policies. Each subject area 
assessment will contain open ended items and assessment of higher 
ord^r reasoning skills will be emphasized. Assessment results will be 
more useful to a varY^ty of audiences as a result , of NAEP adoption of 
new policies. An increased scope of student level and school level 
background information will be collected. The collection of 
background information abou^^^ teachers that can be directly related to 
student achievement has been initiated. Types of information may be 
classified as program related variables, impact of policies and 
practices, effect of teaching strategies and school curricular 
policies related to specific subject areas, and equal learning 
opportunity. (DVffl) 
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What should be assessed? How should it be assessed? As a "national 
report card" on educational progress , the assessment should gather information 
that most Americans consider important. The sk.ills, attitudes and 
understandings assessed should be basic to schooling and measured in ways that 
testing professionals and the public alike consider sound. Further, the 
results must be valid and reliable, both from professional and lay viewpoints. 
Having met criteria for relevance and validity^ the assessment results and 
instruments must also prove useful to a wide range of people dedicated to 
strengthening our educational system. ^ ^ 

With these considerations in mind, the NAEP approach to developing 
assessments rests on a number of principles and assumptions. Primary among 
these are the following:. 

1) NAEP is a public enterprise . It should solicit views from people in 
all walks of life about what should be assessed; the materials and 
processes should be open to public scrutiny; and it should rely on a 
consensus approach to develop objectdves and goals. i 

2) NAEP can impact educational policy . It has an . obligation to address 
issues being raised by major national studies urging educational 

^ improvement. - * , 
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3) NAEP can Impact school and classroom practice >\It can address issues 
related to school effectiveness and improvement and suggest changes 
or actions that might be implemented by teachers and school 
administrators. ' 

4) NAEP methodology should meet the highest technical standards . It 
must continue to pioneer the development of new kinds of items and 
capitalize on the benefits of state-of-the art psychometric theory 
and procedures. ^ <^ ^ 

5) NAEP must learn efficient ways to measure educational progress . It 
must be ^tentive not only to providing more information and 
increasing the utility of that information, but also to conserving 
resources. , * 

To enhance various subject area findings, be more helpful to educational 
practitioners and address policy issues that transcend particular subject 
areas, NAEP has an improved new design. The key features of this desiga 

Q > ' 

include sampling grade as well as age, BIB spiralling and application of item 
response theory. In addition, NAEP has taken innovative steps toward 
improving the utility of information "through the kinds of measures that will 
be included in assessments and the relationships among. those measures. These 
innovations are explained in the following sections. 

NAEP Plans for Selecting Subject Areas 

To provide improved' information about achievement in specific subject 
areas and the relationships in learning and achievement among subject areas , 
NAEP has adopted the following policies: 




• Subject areas will be selected to address Issues of. ij^tlonai!.^ 
' .and infprmation needs, ^ ' 

• At H^st four subject areas will be included in each biennial 
. assessment ' ' ' ' - . 

' • Assessments cff various subject areas will be more closely integrated 
/ and related, _ ; . ' 

, • t. . * '■ ''"^-^ ' . 

• Reading will be included in each biennial- assessment. 'V': 

- Although the selection of subject areas fj3r t!ie current 1983-84 ' 
assessment was completed prior to the NAEP transition to ilS, it should be 
noted that this assessment of reading and writing involves a natural grouping 
of two compatible subject areas that previously had never* been assessed 
together, TJtie way ETS has redesigned the assessment,, spiralling items across 
booklets and IRT scaling, will make it possible for us to investigate 
relationships in learning and achievement among these' important communications 
skills,. 

Communications skills are of current national concern from a human 
resource perspective. "A Nation at ELLsk" states, "The teaching of English 
should equip graduates to: a) comprehend what they read, b) write 
well-organized, effective papers,, and c) listen effectively and discuss ideas 
intelligently." 

Although NAEP will not be able to address listening skills, through a 
variety of reading comprehension items * a variety of open-ended writing tasks 
and tasra that require both reading and writirig, NAEP will be able to answer 
questions about levels of comprehension, effectiveness of written 
communication and how well students can discuss their ideas in writing. The 
1983-84 writing assessment includes more writing tasks than in the past — 15 



exercises at each age/grade level — Covering a variety of informative, 
persuasive and literary writing purposes and situations. In addition, as part 
of the reading assessment students are asked to interpret, analyze and 
evaluate passages in an open— ended f prmat • ^ 

We are currently, engaged in a massive open-ended scoring effort of over 
200,000 responses that are BIB spiralled through the 1985-86 booklets and 
include responses from reassessed items administered in previous assessments, , 
These results, along with the information from the objective -reading 
exercises, should permit us to report levels of performance, how levels of 
performance have changed over time, and for this, assessment , the relationships 
among particular reading and writing skills and abilities. 

NAEP will assess Mathematics, Reading, Science and Computer Competence in 

the 1985-86 assessment. We refer to this assessment as the "Technology 

Package," since it includes two conceptual areas underlying most technologies 

and one important technological application.. As an * integrated assessment 

••package" .mathematics, science and computers complement each other very well, 

and this assessment will enable NAEP to investigate the relationships in 

learning and achievement in these subject areas. 

i ■ . ■ 

In addition, information about these subjects is extremely relevant to 

policy concerns, both separately and in concert. Many major national reports 

about the quality of education in our country have raised questions about the 

level of science and mathematics achievement in our schools and our nation's 

ability to meet the requirements of a technological society. 

Mathematics is a national concern, both as a basic functional" skill and 

as an area necessary to our country's economic growth. For example, between 

1975 and 1980, public fbur-year colleges were forced to increase their 

remedial course offerings by 72 percent and many school districts are in the 



process of increasing mathematics graduation requirements. Similarly, there 

is widespread concern thatf we,' as a country, are riot ^^ell prepared in the area 

of sciencp. Data jcollected by"t;he National Academy of Science indicate tWat 

science study in Japan, Russia, China and we stfe Germany begins in the six£h 

grade. Their report notes, "The time sjp^irit on science, based on class hours^ 

^ ^ . ' 

is approximately three times that s.pent by even the most aijjdence oriented 

students in the United States." Computen^ competency is extremely policy 

relevant because. of the economic advantages* for people with these skilli', the 

.burgeoning presence of computing and -computer courses in schoolsi'^and the ~ 

concerns of^^ business* and Industry. Further, the 15th Annual Gallup Ppll of 

the Public 's"" Attitudes Toward the' Public Schools shows that 80 percent of ^the 

parents with children in schools thap do not have computers say 'they wouldi 

like compAe'rs available for theic children. . * 

Given the current national concerns about economic leadership in 
tomorrow's .worlds the 1985-86 combination of assessments has the/ potential for 
^enormously valifable analysi'^ and reporting* Not only do science and computer" 
competence complenient mathematics and teading to create ^unified as3essment, 
but each' is also individually powerful and gives NAEP the opport^inity ' to be 
innovative and engaged in a "cutting-edge" effort to provide the most 
^enlightening and useful information possible. 2? 

Since reading performance, more than any other area of achievement, is 
considered an indicator 'of student success. and readiness for more advanced 
study in otfhef .fields, | reading will be Included, in each biennial assessment to 
provide an important ^arometer of national ecfucatlonar progress. Frequie'nt <r 
reading assessments will benefit' both the analysis and reporti-ng of NAEP data 
by increasing the number of data points and helping NAEP establish a firm 
trend in performance. NAEP has failed to hold the public's attention in the 



pastr partially because assessments, in an/^given subject matter, have been 
too -Infrequent and sporadic. Rei)orting performance results every two years in 
at least one subject area will give the pjiblic some regularly recurring 
information. 

^ In addition, with the new spiralling design,* student reading performance 
can be linked- to student performance in all the subject areas in each 
assessment. ThiS' link between reading and other subject areas in each 
assessment also will enable NAE^ to relate performance in different subject 
areas assessed in different years. ^ ^ 

Finally, assessing reading will permit a routine analysis of alternating 
cohort samples. The 1983-84, 1987-88 , and 1991-92 samples would represent one 
.set of birth-year cohorts, whil6 the 1985-86, 1989-90, and 1993-94 ^samples 
would represent" an alternate wave 6f student cohorts. With^ reading common to 
all waves,, cohort differences, can be appraised and calibrated. 

In summary, it should be noted that NAEP plans for policy responsive 
s^ection of integrated assessments including four, or more subje^ct areas will, 
of course, continue t;o fulfill the 1978 legislation authorizing NAEP which 
mandates that NAEP "collect, and report at least once every five years data 
assessing the performance of students at various age or grade levels in each 
of the areas of reaciing, writing, and mathematics." As explained previously, 

plans call for reading to be assessed every two years. Also plans are to 

^ ■ ■ ■ * 

assess mathematics and writing every four yearsJ^ Thua, the assessment 

schedule' yields the following picture: ^ . 

Re ad ing Wr i t ing 

Reading Mathematics - Science Computer 
Reading Writing ' ? ', ? 

Reading Mathematics ? ? ' 



1983-84 
19jB5-86 
1987-88 
1989-90 
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The above strategy appears to create a number of attractive Alternatives 

for future, assessments. Many integrated assessments about relevant concerns 

can be developed around the fcore of reading and writing or reading and 
mathematics. , "iX , 



NAEP Plans for Innovative Exercise (Item) Development 



Tp insure that 'NAEP will continue to provide the information^ most 
relevant to educational policy-makers and practitioners, NAEP has adopted the \ 
following policies regarding exercise development: 

• Each subject-area assessment will continue to contain open-ended items 
Emphasis will be placed on assessing higher-order reasoning skills. 

• Research' will be conducted regarding the ef ficiency measures, both 
in terms of the potential for administation techniquesNand the 
economies of measurements that eventually will be gained from IRT 
scaling. , * 

The temptation to save money by developing assessments consisting only of 
multiple-choice items will be staunchly resisted in light of problems with 
face validity and an inability to assess many complex skills. 

As part of the development effort for the 1985-86 assessment, staff is 
implementing a planned, focused effort to assess higher-order reasoning 
competencies, both within and across subject areas. NAEP's ability to provide 
such information will address issues of major concern to educators, business 

C ' • ■ " 

leadeM, ^ and* parents. 

Some higher-ordiar skills and abilities may be j»«bject-area specific, such 
as understanding the relationships underlying the key concepts within each 
specialized discipline; while others, such as pattern recognition , formulating 
hypotheses, applying concepts and generalizations to new situations and 
evaluating information, are. skills that cut across subject areas in ' 
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interesting and meaninigful ways. For example. Academic Preparation for 

College ; What Students Need to Know and Be Able^to Do published by the 

College Board, states that students should be able to "draw reasonable 

conclusions from information found in various sources, whether written, 

spoken, tabular, or graphic/*-^ ^is could include tasks such as interpreting 

written works (reading), interpreting data from maps (social studies), 

generating hypotheses from experimental data (science) and interpreting data 

from tables and graphs (mathematics). ^ , 

The NAEP tasks will relate to knowledge and concepts relevant to subject 

areas. A "content free" thinking assessment containing mental games or 

gymnastics is not envisioned. Rather we hope either to provide students with 

» 

the knowledge and information necessary to perform the given task or to 

construct situations whereby we can determine whether lack of success probably 

resulted from not having the information or from not having an^ appropriate 

^rategy to solve the problem. 

NAEP recognizes that such an endeavor is very difficult and must be 

planned and approached carefully. However, NAEP is equally certain that 

exercises should require students to actually demonstrate their thinking 

skills. For too long,, testing in general has relied on asking students to 

react, rather than act. Of course, this does not mean that NAEP assessments 
t ' ■ 

will not continue to^inclWe the more traditional exercises measuring 

/ \ 

knowledge and attitudes. It is that priority in new development efforts will 
be given to higher-order reasoning e^cercises, particularly those that measure 
Skills that apply across subject areas. 

Work towards this goal began early this year by convening a combined 
meeting of the four Learning ^ Area Committees (^jiading. Mathematics, Science 
and Computer Competence). At a joint session members of all four groups were 



given the same charge — to focus on higher-order skills in guiding the design 
of each particular assessment and then to work together to investigate the 
commonalities in ass^^sb^nts that might permit developing exercises across 
areas* 

There was agreement about the gfbal, and the groups worked independently 
and effectively, basing their definitions of subject-matter domains on 
awareness of current research and their own experience. Each developed, along 
with subject-matter content definitions, a dimension responsive to assessing 
higher-order tasks. Although, these are in draft stages and subject to 
further refinement, the mathematics group suggested assessing the cognitive 
levels of understanding and comprehension, routine applications and problem 
solving/ reasoning. The science team developed a cube using three broad 
terms — content, context and- cognitive levels— ^th the various cognitive 
levels being knowledge, ability to use knowledge and ability to integrate, 
sjmthesize and apply knowledge. The reading experts suggested the 
identification of information, integration/ use of information, extension of 
information and making critical judgments. Finally, the computer committee 
worked in two areas of major concern, applications and computer science. The 
cognitive levels currently proposed for applications are knowledge, operation 
and use; while those for computer science are knowledge, skills, 
understanding, routine application and prpblem/solving reasoning. 

As noted, these initial definitions are currently undergoing a stringent 
review process and the challenge of representing these objectives with valid 
exercises itill lies ahead. Meanwhile, NAEP is searching the literature, 
consulting With experts and exploring a variety of administration techniques 
including' individual or small-group assessments involving "hands-on" 
exercises. In addition, we are investigating ways to capitalize on NAEP 's use 
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of trained exercise administrators and to Implement more Innovative methods of 
open-ended assessment based on NAEP's vast experience and history In ^. this area. 

Finally, NAEP must engage In a planned, focused research effort over t\ie 
next assessments that will enable us to determine which assessment methods 
will most effectively and efficiently address various cognitive levels. It 
may be that a change in emphasis is' needed in that NAEP has not utilized 
resources ef^^ectively and has used expensive methods to collect information 
about achievement related to recognition and recall or even some abilities 
related to higher-order skills that, could be more efficiently and- just as 
effectively measured^ through carefully designed objective measures. » 
Meanwhile, we have neglected a concerted- effort to, evaluate the questions we 
ask students in terms ot the nature of the cognitive demands required. NAEP 
is committed to giving this effort in hopes of providing Improved information 
about students*' thinking abilities. We know it will not be easy and field 
tests may yield disappointing results. Yet, the hope is to advance both the 
art of measurement and the utility of NAEP Inf oirraatlon, 

NAEP Plans for Background Information ^ 

- . V 

To make NAEP more useful to a variety of audiences, assessment results 
must address Issues central to improving the quality of education.* In 
addition, it makes sense that the most useful information about these Issues 
would go beyond describing or explaining difference^ in achievement, and 
suggest changes or actlojxs that might be Implemented. Thus, NAEP has adopted 
the following policies:- x 

• NAEP has substantially Increased the scope of student-level 
background informatj-on collected../ 



• NAEP has 8ub8taA:ially increased th^ scope of scihool-level 



5,backgrbund information collected. , ^ 

• NAEP has initiated the collection of ^ background information 

about teachers ^that Tcan be directly tied to" student achievement. 

r> ' . \ ' ' / 

The types' of information collected may be classified ' into four bro^d 

■ ' ' ■ •■ , I'. 

areas: 1) program-^related varlWW.«s , 2) the impact of policies and' practices 
suggested by the school ef f ectiv6iaess research, 3) the efJEect of teaching 
strategies and school curriciular policies aig^ environments .related to 8peci^*i 

subject areas , and 4) equality of opportunity to learn; , ■ \ ^ 

• " ' 

More specifically, ut^ility of NAEP achievement data is^ being adilressed 
- • ' . V ' • . ■ ■ . 7 

through collected. information about a variety of ^tUdent, classroom/ teacher 

and school variables related to: w ' 

1) the impact of participation in Ifafious programs ;on , 
\ achievement (f or-^example, phaptej'I, bilingual 

education^ programs for handicapped students, programs 
for^ handicapped students, vocational education and 

gffted and talented programs); ^ ^ 

< ' '■ ■ . ^ ' . ' -1 ^ ■ ■ , ■ . 

2) the impact of home on achievement (for^ example, reading 

materials in the home,'> reading *and wriMng activity in 
the home^^Jiom^work done, television viewing,, computer 
access and use, woi^ng mother , family size and . 
Composition, language spoken in the home, level o'f 
* parents' education, after-school supervision and f ree 
. time activities); ' ^ ^ 

3) the quality of the education work force (teacher ^^ 

certification, teacher, training, role of the principal, 

■ > 

size of staff, morale of staff, experience 'of st^ff and 
in-service training opportunities) and that relationsh^^ * ^ 



'with achievement; 
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4) the effect of various school problems (absenteeism, 
discipline, students transfers, vandalism, lafck of 
commitment and motivation and student use of drugs or 
alcohol) on achievement; 

5) the time available for instruction (length of school 
year, day attd c2mb periods) fi^nd that relationship to 
achievement; " 

" 6) the relationship between ethnic composition of^ schools 
and student achievement; 

7) the relationship between student performance and minimal 
competency testing programs or graduation test 
requirements; 

8) the effect of resources as evidenced by pupil-teacher 
ratios, class size, teacher class load, or various 
equipment and facilities on achievement; 

9) how particular curricular approaches (materials, 
textbooks, delivery systems, homework assigned and 
evaluation techniques) relate to student performance in 
particular subjects; . ' 

10) the relationship between time spent in school on the subject 
(courses taken, class time spent and frequency of reading and 
writing activities) and student performance in the subject; 

11) the effect of overall curriculum focus (courses taken, classes 
enrolled in and overall program) on achievement; 

12) the relationship between preschool experience and student 
performance; 
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M3) the relationship between students' perceptions and attitudes 
toward education, school and specific subject matter and 
performance; 

14) how time spent outside of class on subject matter-related 
activities appears to relate to achievement; 

15) how school characteristics may relate to student performance 
(region and size and type of community in which the school is 
located; the kind of school, e.g. public/ private; range of 
grade levels included and the size of school and number of 
faculty); and 

16) how student performance might be influenced by school policies 
regarding ability grouping, remediation, special services. 

In summary, NAEP should be thought of not merely as a social indicator, 
but as a tool to identfy problems and suggest areas of research concerning 
educational progress. Timely analyses of achievement data in relation to 
relevant background variables. should suggest provisional interpretations and 

promising leads that merit further investigation and research. 

\ 

' NAEP Plans for Additional Assessment Activities 

In addition to improved information resulting from timely and integrated 
subject-area assessments, innovative exercise development focused on 
higher-order reasoning skills and the inclusion of an increased number of 
policy-relevant background variables within the context of the existing 
assessment framework, NAEP will seek to conduct special assessments or probes 
in the years between regularly scheduled assessments. 



The first of these activities will be an assessment of young adults, ages 
21-25, to provide a profile of the literacy skills of these adults including 
both high school <!^opouts and college graduates. This additional grant from 
NIE marks a return of NAEP to an original goal of the program, assessing young 
adults. 

This group is particularly critical since most are about to assume 
responsibility for the nation's work. As recent products of our education and 
training programs, they are still young enough to maximize the benefits of 

) 

corrective remedial efforts. 

. i 

The project will build upon recent efforts to address adult literacy 
issues by i:eviewing current work and holding two conferences to help reach a 
consensus on the definition of "literacy," set objectives and select sample 
exercises. As with tlie in-school assessment^ particular care will also be 
given to identifying the background characteristics and educational 
experiences associated with various levels i.of literacy. 

The data generated should provide information to policymakers and program 
operators concerned with providing young adults with the reading, writing, 
computational, problem-solving and work awareness skills needed by individuals 
and society for social and employment purposes. 

NAEP's Process for Developing Assessments 

It should be stressed that NAEP in its effort to address new issues and 
provide more useful information will rely on an assessment development process 
that will be, if anything, even more rigorous than in the past. In addition 
to careful attention to the use of numerous external consultants representing 
various diverse perspectives, backgrounds and constituencies, we will use 
empirical techniques to determine the effectiveness of each item. 



The legislation that created NAEP is clear in its statement that 
objectives should be arrived at through a consensus process, and our intention 
is to follow that specification scrupulously. The procedures followed for 
determining objectives for assessments are essentially those followed by NAEP 
in the past. Objectives from previous assessments were reviewed by a wide 
range -of educators and specialists in the field selected to represent • 
differing points of view, geographical locations, backgrounds and educational 
environments. With the gAiidance of the 'learning Area Committees, selected 
again with great care, the results of reviews were synthesized and objectives 
were modified and updated. ' This new edition of the objectives will in turn be 
reviewed by practitioners and members of the lay public from around the 
country for their reactions and opinions. Several redrafting and review 
phases will be conducted, with the Learning Area Committees participating in 
each phase of the process. 

Similarly, a carefully tested series of steps essentially those followed 
by NAEP in the past, will be used to develop exercises. Based on the revised 
objectives, existing assessment items, previous assessment data and past 
experience, NAEP staff and external consultants will develop exercise 
prototypes and specifications. Exercises will, for most part, be developed 
during item writing workshops and conferences conducted by staff and involving 
external consultants. The items that have been created will be reyiewed and 
revised by staff and by external reviewers considered subject-matter and 
measurement experts. In addition, all exercises will be reyiewed for bias. 
Further language editing and sensitivity reviews will be conducted following 
ETS quality control procedures and all materials will be cleared by NIE, FEDAC 
and 0MB. National field tests will be conducted and the results scored and 
analyzed. Based on the results of field tests the entire process of 

16 
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modifying, reviewing and editing exercises will be repeated. With the help of 
staff and outside reviewers, the Learning Area Committees will select the 
specific exercises that will be included in the asses.sment, 

) 

nary 



NAEP will now assess at least four subject areas every other year. By * 
selecting relevant and related subject areas, this will cost less and give 
NAEP the power to. examine relationships across subject areas. As the NAEP 
legislation mandates, assessment objectives will* be developed through a 
consensus process that meets the needs of the public and respects NAEP's 
public-funding status. The new assessments will present a series of exerci$es 
that can be analyzed to measure of number of learning factors in various ^ 
combinations and examine the interrelationships ^mong questions. Development 
efforts will focus on innovative open-ended assessment of reaisoning skiilis. 
However, to remain relevant and efficient, NAEP must find the resources^ for 
expanded assessment opportunities through continual redesign andi , 
implementation of more cost-effective procedures. 

Increased attention to the collection of student, school and 
classroom/teacher variables from students, school officials and teachers will 
enable NAEP to address current national concerns that focus on school 
effectiveness questions, program policies, teaching strategies and equity 
issues. All development procedures will be conducted to meet high technical 
standards. 
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